build: enable Spark SQL tests for released Spark 4.2.0 - #4950

Inherited from 4.1.1.diff during reconciliation. The Spark build was
trying to resolve comet-spark-spark4.1_2.13 instead of the 4.2 artifact,
causing the sql_hive jobs to fail before any tests ran.

Spark 4.2 removes GeographyVal / GeometryVal and replaces the getGeography
and getGeometry accessors on SpecializedGetters with a single getBinaryView
returning BinaryView. The shim therefore can no longer be shared between 4.1
and 4.2 out of the spark-4.1+ source root: the major, minor-plus and minor
shim directories are all added as compile source roots, so a spark-4.2 copy
would be a duplicate class rather than an override.

Move the trait into per-version spark-4.1 and spark-4.2 copies.

Point spark.version at the released 4.2.0 instead of 4.2.0-preview4.

Spark 4.2.0 bundles its own copy of
org.apache.datasketches.memory.internal.ResourceImpl inside spark-catalyst,
which collides with the transitive datasketches-memory jar and fails the
maven-enforcer BanDuplicateClasses rule, so ignore that class.

Keep the test-scope Jetty pin at 11.0.26. Spark 4.2.0 ships Jetty 12, but the
Iceberg REST catalog test helper needs jetty-servlet, which Jetty 12 replaced
with jetty-ee10-servlet.

Reseed the Spark SQL test overrides from 4.1.2.diff against the v4.2.0 tag and
drop the 4.2.0-preview4 diff.

The notable reconciliation is the test harness refactor: Spark 4.2 shrinks
SQLTestUtils to a deprecated empty alias and moves withSQLConf,
stripSparkFilter and the test() override into QueryTest, with
SharedSparkSession now extending QueryTest directly. Comet's hooks
(isCometEnabled, the IgnoreComet skip and the stripSparkFilter Comet cases)
move to QueryTest accordingly. isCometEnabled has to qualify
classic.SparkSession there, since a bare SparkSession in that package resolves
to the unified class.

Regenerated with dev/regenerate-golden-files.sh --spark-version 4.2.

The q2, q5, q54 (v1_4) and q5a (v2_7) goldens are pruned: their plans now match
the shared fallback under the released 4.2.0, so the divergences seen on
4.2.0-preview4 are gone.

q77a (v2_7) is added because Spark 4.2 plans a OneRowRelation into its Union
branches. Comet cannot convert that leaf, so the Unions fall back and the
aggregates above them lose their Comet partial producer. The approved plan
records the degraded plan; see apache#4949.

Add a spark_4_2 job to the ci.yml umbrella calling spark_sql_test_reusable.yml
with Spark 4.2.0 on JDK 17, plus the matching spark_4_2 filter in
compute-changes.py and output on the changes job. Gate it behind the
run-spark-4.2-tests label so it runs on pushes to main but stays off the
default PR path while 4.2 support is experimental.

Update the user guide for the released 4.2.0 and refresh the new-Spark-version
contributor guide, which still described the spark_sql_test.yml matrix that was
replaced by the per-version umbrella jobs.

Two Spark 4.2.0 changes surfaced by moving the profile off preview4:

Iceberg scans job: Spark 4.2.0 turned `connector.catalog.View` from an
interface into a class. No Iceberg spark-runtime is published for 4.2, so
the build reuses the 4.0 runtime, whose `SparkView implements View` now
throws IncompatibleClassChangeError at class-load and aborts the Iceberg
suites. Report Iceberg as unavailable on Spark 4.2 in the shared probes so
the suites skip. Guard the fuzz suite's beforeAll with an early return
(cancelling from beforeAll aborts the suite) and add the missing
`assume(icebergAvailable)` to one native-scan test.

Expressions job: Spark 4.2 normalizes NaN / -0.0 for array_distinct and
the array set operations by wrapping their inputs as
`KnownFloatingPointNormalized(ArrayTransform(arr, x -> NormalizeNaNAndZero(x)))`.
The serde only handled a scalar `NormalizeNaNAndZero` child and fell back.
Since `KnownFloatingPointNormalized` is a runtime no-op tag, serialize any
other child directly and let its serde (the ArrayTransform codegen
dispatcher) carry the normalization, keeping these operations native.

# Conflicts:
#	spark/src/main/scala/org/apache/comet/serde/contraintExpressions.scala

CometNativeScan serializes the full data and partition schema, not just
the required columns. A table with a GEOMETRY or GEOGRAPHY column (Spark
4.2) has no proto representation for those types, so schema2Proto threw
NoSuchElementException during planning instead of falling back. Reject
the native scan when any data or partition schema field type cannot be
serialized.

…issues

Handle new-in-4.2 Spark SQL test suites that assert on Spark-internal
execution mechanics Comet replaces wholesale, and repoint deferred
assumes at specific issues instead of the closed generic tracker (apache#4142).

- SLAM metric suites, segment-tree window suites, and the UnionExec
  whole-stage-codegen test are marked IgnoreComet / IgnoreCometSuite
  (apache#4963, apache#4964, apache#4965).
- collect_set NaN/-0.0 normalization deferred (apache#4966).
- Repoint assume(!isSpark42Plus) guards to specific issues: apache#4967
  (ANSI arithmetic), apache#4968 (BloomFilter), apache#4969 (Iceberg REST catalog).
- Geometry/geography scan tests now pass via native-scan fallback.

`PR Build [expressions]` (Spark 4.0 / 4.1 / 4.2) failed on
`expressions/misc/variant.sql`: the new full-data-schema serializability
check in `CometNativeScan.isSupported` ran before the projected-schema type
check, so a `struct<v: variant>` column was reported as "Native scan does not
support data type struct<v:variant>" instead of the expected "type
VariantType". Run the schema-support check first so the more specific type
reason wins; the geospatial case that motivated the check still falls back
(and no longer crashes planning), now covered by a new
`expressions/misc/geospatial_types.sql` fixture.

`Spark SQL Tests (Spark 4.2)` sql_core failures:

- Spark 4.2 allows `FILTER (WHERE ...)` on window aggregates. Comet dropped
  the filter (it is only serialized for Partial mode aggregates, and window
  aggregates are Complete mode) and returned wrong results for `window.sql`.
  Decline these window expressions in `CometWindowExec`, and turn the silent
  drop in `aggExprToProto` into a fallback for any non-Partial mode carrying a
  filter. Covered by `expressions/window/window_filter.sql`.
- `dev/diffs/4.2.0.diff` was reseeded from `4.1.2.diff` and missed the updates
  main has since made to the 4.1 diff, which caused the `SubquerySuite`,
  `CachedBatchSerializerNoUnwrapSuite` and `DataFrameAggregateSuite`
  (x4 suites) failures. Ported: the `CometHashAggregateExec` arm for
  SPARK-22223, the `WholeStageCodegenExec(CometColumnarToRowExec(...))` scan
  shapes for SPARK-26893, the codegen-aware cached-plan transition check, the
  `VariantEndToEndSuite` patch, `spark.comet.shuffle.enabled` for the renamed
  config, and `comet.version` 1.1.0-SNAPSHOT.
- `tuplesketch.sql` hits the same collated-string sketch problem as
  `thetasketch.sql`, so it joins the ignore list. `join-nearest-by.sql` records
  Spark operator names in EXPLAIN goldens and `in-order-by.sql` orders by a
  column with ties, so both run with Comet disabled (as explain*.sql and
  in-limit.sql already do).

Also merges apache/main, which the branch needed for the diff sync.

apache#5881 added `.github/actions/maven-bootstrap/**` to every Spark SQL job's
change filter. `spark_4_2` exists only on this branch, so that commit could
not reach it, and merging main produced no conflict: the entry was appended
to five sibling lists and the sixth simply never grew one.

check-ci-config.py catches this -- `spark_4_2` is in BUILD_JOBS, so the
maven-bootstrap routing case expects it -- and it fails on the merge commit
as it stands, which would take preflight down with it.

4.2.0.diff was seeded from 4.1.3.diff and last reconciled against main on
2026-08-31. Seven commits have changed 4.1.3.diff since, and none of them
could reach 4.2.0.diff: the file exists only on this branch, so every one of
them merged cleanly while leaving the 4.2 diff on the old behaviour.

Ported, each regenerated from a v4.2.0 checkout rather than hand-edited:

- apache#5755, apache#5760, apache#5745, apache#6046 restored coverage that Comet had since fixed.
  4.2.0.diff still carried the skips: the recursive HAVING/ORDER BY fixture,
  the DELTA_LENGTH_BYTE_ARRAY test and `withAllParquetWriters`, the two
  Variant shredding suites, and the invalid-row-index-column `assume(false)`.
- apache#5914 added `--SET spark.comet.enabled=false` to two ORDER BY fixtures
  whose tied rows have no deterministic order.
- apache#5987 mixed IgnoreCometSuite into the RocksDB state-store suites. Five were
  missing entirely; RocksDBStateStoreIntegrationSuite had the mixin last,
  which skips only the plain `test(...)` registrations, so it moves ahead of
  AlsoTestWithRocksDBFeatures like the others.
- apache#5821 replaced the direct `isInstanceOf[EmptyRelationExec]` assertions in
  AdaptiveQueryExecSuite with an `isEmptyRelation` helper that also accepts
  CometEmptyRelationExec.

Two 4.1.3 hunks are deliberately not ported. AdaptiveQueryExecSuite weakens
`localReads.length` to 1 inside two tests that 4.2 already annotates with
IgnoreComet, so on 4.2 the assertion is unreachable either way; leaving
Spark's own value there keeps the diff smaller.

Verified: applies with no rejects to a pristine v4.2.0 worktree, and every
file the two diffs share now carries the same Comet annotations apart from
the documented 4.2 reconciliations (the SQLTestUtils-to-QueryTest move, the
extra BroadcastHashJoinExec parameter, and the 4.2-only sketch fixtures).

The 4.2 suite was left on demand -- the `run-spark-4.2-tests` label or a
dispatch -- on the grounds that a brand new version should not gate anything
until it settles. It still gates nothing in the nightly tier: a red nightly
opens a `ci-nightly-failure` issue and blocks no merge.

What on demand actually cost is the 4.2 diff file. Every other version's diff
is updated as a side effect of its suite running; 4.2's was updated only when
someone asked for the suite, and it spent three weeks silently behind the
others. Nightly is what keeps it current.

`spark_4_2` moves from its own SPARK_EXPERIMENTAL set into NIGHTLY_TIER,
which is also what makes the `schedule` POLICY case assert its presence
rather than its absence. The label keeps working and now means the same thing
it means for 3.5 and 4.0: bring the nightly run forward onto this pull
request.

…ders

apache#5914 replaced the `replacing-missing-expression-with-alias.sql` and
`in-set-operations.sql` entries in `SQLQueryTestSuite.ignoreList` with
`--SET spark.comet.enabled=false` headers in the fixtures themselves, and
closed apache#5570. The previous commit carried the headers into 4.2 but left the
`ignoreList` entries in place, which changes nothing: `createScalaTestCase`
consults `ignoreList` before any Comet-specific dispatch, so both files were
still registered as ScalaTest `ignore` and skipped in the `ENABLE_COMET=false`
baseline too.

Removing them leaves 4.2 with no Comet additions to `ignoreList` at all --
the sketch files already moved to fixture headers in e0c7963 -- so every
`.sql` fixture now runs in the baseline, and the two ordering-sensitive ones
run with Comet disabled rather than not running.

apache#6085 rewrote the tier section of ci.md around a mermaid diagram and a
suite-by-suite table, both of which landed while this branch had 4.2 on
demand. Adds 4.2 to the nightly row of each, and `run-spark-4.2-tests` to the
opt-in label table.

# Conflicts:
#	spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala

apache#6066 removed memory accounting from Comet's on-heap mode, which the Spark SQL
jobs run in, and dropped this setting from the other version diffs. 4.2.0.diff
only exists on this branch, so the merge left it behind.

…alid UTF-8

hll.sql, thetasketch.sql and tuplesketch.sql each ran every query with Comet
disabled because one table in each file holds invalid UTF-8, which Comet's
native scan rejects (apache#4121). Only the eight queries per file that read that
table fail, so the rest of each file lost Comet coverage for nothing.

SQLQueryTestSuite now disables only the native scan, and only for queries that
name hll_string_test or t_string_collation, and the three fixture headers are
gone. Doing it in the harness rather than with SET statements in the fixtures
keeps the diff free of golden-file changes.